A service is a set of code that exposes an application programming interface (API). Another set of code transmits a request to the API, to perform one or more operations indicated by the request. The set of code that transmits the request may be referred to by many names, such as user, consumer, requesting entity, etc. As one example, a web service exposes an API for requesting web pages. A web browser (the requesting entity in this example) transmits a request to the API over the Internet. In response to the request, the web service transmits a web page (e.g., hypertext markup language (HTML)) to the web browser.
Many services operate according to a synchronous model. In a synchronous model, the requesting entity maintains a persistent connection to the service for the duration of the request-response cycle. To process the request, the service may also maintain at least one thread that persists through the duration of the request-response cycle. In the web service example above, a single thread may execute code that receives the request, retrieves the appropriate HTML from a server, and transmits the HTML to the web browser.
Synchronous service models may have various performance concerns. For example, if a requesting entity is forced to maintain a persistent connection with the service throughout the request-response cycle, the requesting entity may effectively be precluded from performing other operations while waiting for the response. As another example, if the service maintains a thread that persists through the duration of the request-response cycle, that thread may consume processing cycles unnecessarily, degrading the web service's performance. Alternatively or in addition, the persistent thread may present a single point of failure, where if the thread fails, the entire request-response cycle fails.
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.